Optimized Virtual Channel Assignment in the Cray XT

The Cray XT is an MPP system that scales up to 32K nodes using a bidirectional 3-dimensional torus interconnection network. Four virtual channels are used to provide point-to-point flow control and deadlock avoidance. Using virtual channels avoids unnecessary head-of-line (HoL) blocking for different network traffic flows, however, the extent to which virtual channels improves network utilization depends on the distribution of packets among the virtual channels. This paper investigates the virtual channel balance—relative traffic carried on each virtual channel– and its importance on network utilization. We discuss the routing algorithm and use of virtual channel datelines to avoid deadlocks around the torus links, and heuristics to balance the packet load across the virtual channels. We present network performance results from an 11×12×16 3D torus network.